Repository navigation
fix(windows): allow multiple in-flight tap writes to stop routing stalls (#1428) - #2635
Open
NullP0interEx wants to merge 1 commit into
Open
NullP0interEx wants to merge 1 commit into
NullP0interEx wants to merge 1 commit into
Conversation
The tap driver keeps every IRP_MJ_WRITE pending until NDIS returns the indicated NET_BUFFER_LIST (rxpath.c, ReceiveFlags 0). WindowsEthernetTap only ever had one overlapped write outstanding, so a single frame that the Windows stack holds on to blocked every following inbound frame. The stack holds a frame for up to ~3 s when it forwards it to a directly connected address it has to resolve via ARP first (1 s per attempt, up to 3 attempts). On a Windows host that routes between ZeroTier and a LAN this stalls all ZeroTier traffic of that host for 1-3 s whenever something reaches for an IP that does not answer, and repeated requests stack up into a complete outage. Keep up to ZT_WINDOWS_TAP_WRITE_SLOTS (16) writes in flight, each with its own OVERLAPPED and buffer, so a held frame only occupies its own slot. Frames are still submitted in queue order; the driver indicates each one synchronously inside IRP_MJ_WRITE, only the completion is deferred. On shutdown wait for outstanding writes before the buffers go away, as the driver marks these IRPs non-cancellable. Fixes zerotier#1428
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1428.
Problem
On Windows hosts that forward traffic between a ZeroTier network and a LAN, all inbound ZeroTier traffic of that host stalls for 1 to 3 s every time something reaches for a LAN address that does not answer ARP. With enough such traffic the stalls overlap and the ZeroTier interface stops passing traffic completely. #1428 describes this in detail.
Root cause
The analysis in #1428 suspected the driver's NBL handling. The driver does play a part, but the stall comes from the combination of driver and service:
windows/TapDriver6/rxpath.c,IRP_MJ_WRITE): every frame the service writes is wrapped in an NBL that points straight at the IRP's MDL. The driver callsIoMarkIrpPendingandNdisMIndicateReceiveNetBufferLists(..., ReceiveFlags 0). The IRP is only completed when NDIS hands the NBL back throughMiniportReturnNetBufferLists. WithoutNDIS_RECEIVE_FLAGS_RESOURCESthe stack may keep the NBL as long as it needs.osdep/WindowsEthernetTap.cpp,threadMain): the tap I/O thread has exactly oneOVERLAPPEDfor writes and onewriteInProgressflag. While that one write is pending, the loopcontinues and nothing else from_injectPendinggets written.Together: one forwarded frame to a dead LAN IP holds the single write slot for up to 3 s. Every other inbound frame, including unrelated traffic to the host itself, waits behind it in
_injectPending. The encrypted packets arrive at the physical NIC on time, but the ZeroTier adapter delivers them seconds later. In the captures below the gap is up to 7 s.Fix
Allow up to
ZT_WINDOWS_TAP_WRITE_SLOTS(16) writes to the tap device at once:OVERLAPPED, event and buffer. The frame is copied from_injectPendinginto the slot's buffer beforeWriteFile, because the driver keeps referencing that memory (via the MDL) until the IRP completes.hEvents of the slots in flight, plus the inject semaphore and the read event as before.NdisMIndicateReceiveNetBufferListssynchronously inside theIRP_MJ_WRITEdispatch routine, so frames reach the stack in the order they were written. Only the IRP completion is deferred.GetOverlappedResult(..., TRUE)before the buffers and events go away. The driver marks these IRPs as not cancellable, soCancelIodoes not complete them.A frame that the stack holds now occupies only its own slot, and everything else keeps flowing.
The change is service-only. The driver and its signature are untouched, so it can ship in a normal release.
Remaining limitation
If more than 16 frames to unresolvable addresses arrive within about 3 s, all slots are taken and the old behaviour returns for the rest of that window. A complete fix in the driver would be to set
NDIS_RECEIVE_FLAGS_RESOURCES, or to copy into a driver-owned buffer and complete the IRP right away. That needs a re-signed driver. This PR removes the problem in practice and is independent of a driver change. The slot count is a single define if a larger value is preferred.Test
Setup: Windows 10 Pro 22H2 (19045) workstation acting as the LAN router of a ZeroTier network.
IPEnableRouter=1, RemoteAccess service, no NAT, managed route192.168.3.0/24 viathis host, LAN router with a static route back. ZeroTier 1.16.2, tap driver 3.0.0.1.Build: tag
1.16.2plus this patch, built with VS Build Tools 2022 (v143) and Rust stable,msbuild windows\ZeroTierOne.sln /p:Configuration=Release /p:Platform=x64 /t:ZeroTierOne. Onlyzerotier-one_x64.exewas replaced, the driver was left as installed.Measurement: 120 ICMP echo requests at 0.5 s intervals from a macOS peer to the router's ZeroTier IP. At the time, the router had several ARP entries in
Unreachablestate on the LAN side. The runs alternated between the official binary and the patched one, restarting the service each time:With the official binary the RTTs form a falling sawtooth (7383, 6882, 6377, 5875 ...): the replies are held and then released together. With the patch the RTTs stay at the path's normal 20 to 30 ms. Runs 1 and 3 show that a service restart alone does not change anything.
Before this, a
pktmoncapture on the router showed the encrypted reply reaching the physical NIC 1 ms after the request, while the decrypted frame was only indicated on the ZeroTier adapter 2.9 s later.The patched binary is now also deployed on two Windows Server 2019 VMs (not routing). I will report back here if anything shows up in longer operation.